Object Description Using Visual and Tactile Data

With the development of vision and haptic sensor technologies, robots have become increasingly capable perceiving their external environments. Although machine haptics surpassed humans in some aspects perception, it is difficult for to describe objects from multiple viewpoints by using a combination visual modalities. In this paper, we use convolutional neural networks separately extract features then fuse these two types features. Then, multitask learning combined with multilabel classification form multitask-multilabel method. The developed method used identify color, shape, material attributes, class an object visual-haptic fused feature vector. To verify effectiveness proposed description method, experiments are conducted on PHAC-2 dataset collected VHAC dataset. experimental results show that produces most accurate descriptions smallest number parameters.